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ABSTRACT 

The Continuing Education Institute (CEI) established 
a seven-membei training collaborative to upgra'le the literacy cind 
English Ic^nguag ^ skills of nursing assistants, dietary aides, cind 
housekeeping workers employed in a chronic and acute care hospital 
and in five Massachusetts nursing homes. Three adult basic education 
(ABE) classes, five English-as-a-Second-Language (ESL) classes, and 
one adult degree class met at the Shattuck Hospital during a 15-month 
period, while other classes were held at other sites. CEI worksite 
training programs were characterized by a strong link between course 
content and the trainee's job functions. Counseling cind child care 
arrangements also were made for the employees. Employee participants 
benefited by becoming more skilled in their jobs, preparing for 
training for advancement, and becoming more comfortable socially. 
Employers benefited by using the training as a recruitment tool and 
by increasing attendance and pre iuctivity of class participants. The 
program was regarded as a success by participants and employers. 
(This report includes course descriptions and descriptions of the 
programs as implemented at each site. An external program evaluation 
is appended.) (KC) 
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WORKPLACE LITERACY PARTNERSHIP FOR HUnSING 
EMPLOYEES 



OVERVIER OP THE PROGRAM 

The Continuing Education Institute (CEI) established a seven 
member training collaborative to upgrade the literacy and English 
language skills of nursing assistants, dietary, and housekeeping 
workers employed in a chronic and acute care hospital and in 5 
Massachusetts nursing homes. The partners were Lemuel Shattuck 
Hospital, Armenian Nursing Home (Jamaica Plain), The Greenery 
Rehabilitation and Skilled Nursing Center (Brighton), 
Massachusetts Long Term Care Foundation (Dedham), Vernon Hall 
Nursing Home (Cambridge), and the Westridge Healthcare Center 
(Marlborough) . 

CEI established and operated 3 educational programs: Adult 
Basic Education (ABE), English as a Second Language (ESL), and 
the High School Diploma (ADP) creating 100 training positions for 
employees cf the 6 health care organizations. There were 3 ABE 
classes, 5 ESL classes, and 1 ADP class which met at the Shattuck 
Hospital . 

The three educational programs were funded through the U.S. 
Department of Education and were implemented during a 15 month 
period in addition to a two month planning, or start-up period. 

LITERACY REQUIREMENTS IN A HEALTH CARE SETTING 

CEI worksite training programs were characterized by a 
strong link between course content and the trainee's job 
functions. Ongoing, regular contacts between CEI staff and 
employer supervisors were scheduled to ensure that the literacy 
skills taught related to those required of the trainees on the 
3ob. In addition to ABE and ESL textbooks which have a strong 
emphasis on the world of work, CEI staff have available a wealth 
of teacher-produced readings and practice materials based on 
nursing home and hospital content. 

PoK ^IV"^ trainee benefited from the program in several ways. 
d?™f • t students, the workplace-specific curriculum had a 
direct impact on performance and productivity because of their 
improved oral and written communication abilities. Many train 
in ADP secured the skills need to be accepted for nore 

f^hfiffi"^"^ training, an absolute prerequisite to advancement in 
a health care career 
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sponsoring emploiers also benefited. The possibility of 
paid training was used successfully by some nursing homes as a 
recruitment tool . Nursing home staff who have participated have 
improved their attendance and increased productivity. Employers 
who subsidized and supported training engendered feelings of 
loyalty on the part of employees. Overall retention rates 
increased, as did options for promotions from within. Finally^ 
careful scheduling of courses at the worksite resulted in a 
virtually uninterrupted work flow. 

One of the most important issues regarding the relevance of 
the Workplace Literacy Program to the needs of the health care 
industry relates to the implementation of certification 
requirements for nursing assistants, the result of federal and 
state legislation enacted to improve the quality of care for 
Medicare and Medicaid certified nursing homes. The national 
Nursing Home Reform Act of 1987 required that as of October 1989, 
all newly hired nursing assistants must successfully complete 
within 120 days a minimum of 75 hours of training in a program 
approved by the state, as well as pass a state-approved 
competency evaluation. Nursing assistants employed by a facility 
before July 1989 are required to take the competency evaluation 
within 180 days. Adequate literacy skills are a necessary 
condition for these direct-ca*:e personnel to participate in 
training and to pass the certification evaluation which are both 
based on standards established by the U.S. Department of Health 
and Human Services. 

In order to conduct the 75 hour training program, a RN staff 
member must complete an approved course with a length of 24 
hours. The Train the Trainer Course conducted by Workplace 
Literacy Partnership staff was designed to fulfill this 
requirement. There were 95 participants who attended the course. 



OBJECTIVES OP THE WORKPLACE LITERACY PARTNERSHIP 



The Workplace Literacy Partnership for Nursing Home 
Employees was designed to achieve a number of objectives in the 
sponsoring health care facilities. They ranged from the social, 
which included intangible, but important benefits for residents 
and employees, to the three separate educational programs that 
were created. In addition, the economic impact of the 
Partnership on the nursing homes and on their employees was 
reflected in the set of economic goals. 
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•Those objectives included: 



SOCIAL 

i. enhanced the quality of care and well-being of the 

residents and patients of the health care facilities 
2* fostered inter-generational communication 
3. raised the level of basic education and English 

language skills of women, minorities, and immigrants — 
major segments of the workplace of the future 



ECONOMIC 

1. improved the job performance and productivity of 
employees 

2. facilitated the recruitment of direct-care workers and 
other staff 

3* increased the retention of employees who had 

demonstrated a commitment to geriatric and long-term 
care nursing 

4. helped nursing homes implement training requirements 
mandated by state and federal government 

5. provided the skills for employees to undertake advanced 
training programs 

6. enhanced the earning potential and broadened the career 
options of participating employees 



PROGRAM 

1- provided adult basic education — reading, writing, and 
math -- to 20 employees of 3 nursing homes 

2. established an English as a Second Language Program for 
60 limited English speakers of 5 nursing homes 

3. offered a high school diploma program at Lemuel 
Shattuck Hospital to 10 of its employees and to 10 
employees of 3 nursing homes 

4. helped nursing assistants acquire the literacy skills 
needed to pass state-mandated training and competency 
evaluation requirements 

5. made available a range of support services (counseling, 
tutoring, transportation, and child care) to facilitate 
employee paticipation in these classes 

As a whole, the Workplace Literacy Partnership for Nursinq 
Home Employees enabled the primary learning provider, CEI, to 
make a comprehensive assessment of employee literacy and language 
needs in 5 health care organizations and to service them 
appropriately. 
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COMMITMENT OP FMtVfSRSEl? MEMBERS 



As primary learning provider, CEI was the lead member and 
representative of the partnership. CEI assumed fiscal and 
operational responsibility, as well as provided all education, 
counseling, and related services. 

The health care organizations provided paid release time to 
employees enrolled in the classes. Each facility contributed 
classroom space, and the administrators authorized senior staff 
and department heads to work with their CEI counterparts in 
program scheduling. One member of each facility was designated 
to serve as the liaison for the respective facility with CEI 
managers and instructional personnel. 

Each of the partners had a representative serve on the 
Advisory Board which me<: a total of six times. 



TRAINEE POPULATION 

The most typical trainees were female 'nursing assitants in 
the 25-34 year age range. Prior to their current positions, many 
were out of the workforce or out of school for many years. Many 
more foreign born than native born were represented. Among the 
non-native speakers, the overwhelming majority were Haitians, 
followed by Hispanics, Portuguese speaking, and Asians. Over 
half of the employee group indicated that they had completed high 
school. That number includes immigrants who reported having 
earned high school diplomas in their countries of origin. 
However the educational equivalent of these foreign diplomas is 
difficult to ascertain. 

Despite different ethnic backgrounds, command of English, 
and basic skill level, participants in the Workplace Literacy 
Partnership share several characteristics. They were nominated 
by their employers on the basis of their commitment to a career 
in nursing--especially in a long term care environment— and for 
their excellent attitudes and attendance on the job. Further 
almost all of the native born trainees lacked the reading and' 
math skills that are needed for post-secondary technical 
coursework (high school diplomas notwithstanding). Lastly, all 
lacked the econo; ic independence to support themselves (and 
family members, in some cases) for 12 months and to pay for 
education-related expenses such as tuition, books, and fees. 



C* Tutoring 



When recruitment and assessment are correctly conducted and 
when instruction and materials are geared to the 
appropriate level, the majority of learners are likely to 
meet the course objectives. Some students — especially those 
who had been out of school for many years or who had 
learning disabilities — required assistance beyond what the 
teacher was able to provide. Moreover, even when students 
were progressing normally, tutoring provided reinforcement 
for optimum learning and, with limited English speakers, 
extra practice time. 

Volunteer tutors were recruited from several sources: the 
Commonwealth Literacy Campaign (CLC) and volunteers from the 
nursing home supervisory staff. In other CEI literacy 
programs held at long-term care worksites, supervisory 
stiff, co-workers, and residents have sometimes served as 
tutors, or language coaches. Such support of the trainees 
is a natural outgrowth of the quasi-familial atmosphere that 
characterizes many long-term care facilities. If available, 
this assistance is a welcome addition, and when appropriate, 
instructors consult with them to identify skill areas 
needing extra practice in the tutorial setting. 



DESCRIPTION OF PROGRAMS 



A. Adult Basic Education 

A competency-based curriculum was designed for nursing 
home employees with relatively low literacy skills (below 
4th grade). The ABE course consisted of three 13-week * 
modules, with reading, writing, and mathematics 
integrated. Students spent 52 hours in each module (4 
hours a week for 13 weeks) for a total of 156 hours of . 
instruction. Counseling and tutoring, which were 
described above took place beyond the scheduled class 
hours . 
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The ABE program was offered at 3 sites with the 
following numbers of students: 



SITE 



Registered 



Withdrew 



Completed 



The Greenery 

Armenian 

Westridge 



13 
4 
6 



2 
2 
1 



11 
2 
5 



Total 



5 (22%) 



18 (78%) 



Students attended class on the average of 76% of the time 
with a range of 51% to 100%. 



B. English as a Second Language 



The ESL course sequence consisted of three 13-week 
modules, providing a total of 156 hours of class. The 
structure or the ESL course is similar to that of the ABE 
modules. Classes met twice weekly for two hour periods, 
with each module offering 52 hours of instruction. 
Counseling and tutoring — integral parts of the programs- 
were offered in addition to these class hours. 

CEI's philosophy of ESL instruction is aimed at enabling 
the employees (1) to communicate accurately and 
confidently in English on the job and (2) to acquire the 
lifeskills needed to adapt successfully to living in the 
United States. A CEI language class is lively and 
interactive; student practice is maximized. 

In CEI's nursing home ESL program course content is 
adapted after an analysis of the actual communication 
needed for effective job performance. Instructors 
concentrate on workplace ESL, as opposed to teaching 
general or academic English. The learning provider is 
experienced in meeting the learners* basic educational 
needs --especially ESL--in nursing home settings. CEI 
staff have developed curricula covering the beginning to 
low intermediate level and the intermediate to high 
intermediate level. These curricula were the product of 
on-site analyses of the communication needs of direct- 
care workers. A rich collecion of handouts and 
exercises based on this content was made available for 
classroom use. 
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students fulfilled the remaining requirements for the 
diploma — about a third of the credits needed — in the 
following manner: 

* Informal Life Experience Workshop...,, credits 

* Life Experience Employment Workshop.. 4 credits 



Upon successful completion of the coursework and Life 
Experience Workshops, employees earn a high school 
diploma from Christopher Columbus High School, a private 
accredited institution in Boston. 



The ADP sequence involved 170 to 190 hours of class time. 
The course descriptions follow: 



This course focuses on the critical thinking and reading 



skills prerequisite for completing the other courses in the Adult 
Diploma Program. Topics range from reading skills (pre-reading, 
locating main ideas, supporting details, finding the author's 
purpose, scope and sequence, etc.) to reading for meaning 
(inference, discerning fact from opinion, implied main ideas, 
etc.,). Emphasis is placed on class participation and using 
texts appropriately. 



The CEI writing course emphasizes buildinc a personal 
narrative based on the adult's experience in life. Essays are 
individually created under the careful supervision of the course 
instructors. Skills developed in class are centered on the 
fundamentals of composition: developing good paragraphs, using 
topic sentences and conclusions, using transitional devices and 
using appropriate punctuation and grammar. A writing process 
approach is used to develop the voice of each class member. 



Mathematics 

The CEI mathematics course presents the basic computational 
skill5? necessary to perform the operations needed in everyday 
ii.'ing. The focus of the course is to obtain competency in th#* 



Total 



16 credits 



Reading 



Writing 
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areas of basic computations (addition, subtraction, 
multiplication, and division), map-reading, basic geometry, 
measurement and metrics, and pre-algebra. All program 
participants pass competency-based tests prior to completing the 
course. 



Science 

The CEl Science course emphasizes the fundamentals of earth, 
life and physical science. A strong focus of the course is the 
hands-on experience of working on science laboratory experiments. 
At the conclusion of the course, each individual obtains a better 
working understanding of the science components contained in 
their world. 



U.S. History 

The CEI U.S. History course emphasized the critical 
thinking, reading and writing skills necessary to integrate the 
concepts for an overview of U.S. History from the colonial period 
to the 1990s. Particular emphasis was placed on classroom 
participation through a dialectical inquiry method of 
instruction. 



Life Experience Employment Workshop 

The CEI Life Experience Workshop is dt r ed to have 
students reflect on their current jobs and .. lue a 4-6 page paper 
describing and analyzing at least eight sKills used in their 
jobs. A particularly powerful process in thi.-. course is the 
transformation students go through when they recognize all the 
many skills they have and how valuable these skills are in their 
first work. Elements of composition introduced in the first 
writing course are re-emphasized in this oiie. 

Life Experience Workshop 

1 •11^'^ ^^^^ workshop students demonstrate expertise in applied 
skUls acquired during their lives. Examples include a second 
language m which they are evaluated, a high school transcript 
indicating that two major subjects have been successfully 
completed, demonstrating skills in home management, sports, 
recreation, travel, fine arts, practical arts, family health, 
community service. Students compose a short paper on the 
learning involved in the skill and present their findings in 
C 1 stss • 
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Description of Students 



Eighty percent of the class were nursing assistants. 
Students ranged from 28 to 61 years of age, with an average 
age of 32. Seventy-five percent of the class were women. 
Ninety percent were immigrants from Haiti, Jamaica, and 
Barbados . 



Registered Withdrew Completed 



Of the 20 trainees who undertook the AEP, 13 completed the 
graduation requirements established by Christopher Columbus 
High School. In general, the trainees who undertook the ADP 
read on the 7th-9th grade level. By graduation, their 
average reading levels increased by 3 levels. 



Individualized Educational plans 

In the CEI approach to adult education, individualized 
educational plans are established for each trainee. Broad 
educational/career goals and discrete objectives in the 
various skill areas are identified as a result of the 
assessment process. 

On-site orientation and screening was conducted by the 
Course Coordinators and/or the instructors under the 
direction of the Program Manager. Assessment and placement 
required 3-5 days at each worksite over a period of 2-3 
weeks. The process differed for English dominant and 
limited English speakers. 

All employees nominated by their employers for the ESL 
program were evaluated individually with an oral interview, 
which consisted cf a needs profile, and tests for auditory 
discrimination, sight word recognition, and aural 
vocabulary. Following the oral interview, the CEI Reading 
and Writing Assessment Instruments were administered. This 
exam consists of reading passages and performing grammar 
exercises of increasing difficulty ranging from low - 
beginning through advanced ESL levels. Candidates were also 
given the opportunity to express themselves in a writing 
exercise. 



20 



7 (35%) 



13 (65%) 
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Individuals who were not able to take this exam because 
their literacy levels were too low were given the CEI 
Literacy Assessment., which consists of letter, number and 
sentence recognition and comprehension which places 
candidates in either ESL Literacy Level 1 or 2. 

The trainees also took pre- and post-tests for each module 
of instruction so as to provide quantitative, as well as 
qualitative, monitoring of progress. 



In individual interviews, the ABE, and ADP candidates had 
the opportunity to discuss their school history and 
acaoemic/employment goals. Depending upon their educational 
background; applicants took several of the following 
measures: the Wide Range Achievement Test, the Slosson Oral 
Reading Test, California Adult Student Assessment Systems, 
CEI Informal Reading Inventory, CEI Math Assessment 
instrument, and CEI Writing Assessment Instrument. The 
tests were administered in small groups of 3-4 candidates to 
minimize interference with the employees' work schedules. 
In addition to interviewing each interested employee, CEI 
staff met with supervisory personnel to identify the kind 
and level of basic skills that were needed on the job. 

Information derived from these activities served as the 
basis of the individualized educational plans. ESL and ABE 
students met with their instructors and with their 
coordinators on a regular basis to review their progress 
and/or raise concerns. The individualized plan was 
frequently amended as a result of these meetings. Depending 
on specific academic strengths and weaknesses 
tutorial assistance was sometimes required. Although 
instructors offered individualized instruction in class and 
through homework assignments, tutors — by definition — 
served ?s the greatest source of individualization. 



D. TRA I N-THE -TRAINER SEMINAR 



m 



A 24 contact hour training course was developed m 
accordance with state and federal requirements for instructors of 
nursing assistants. This course was approved by the Massachi'<^etts 
Department of Public Health. The 102 member audience was 
comprised of registered nurses with at least one year's 
experience working in a nursing home, hot with less than one 
year s teaching experience. Non-nursing nome employees planning 
to teach nursing assistants were allowed to participate if they 
had two year's experience in the care of the elderly or 
chronically ill, as well as a year's teaching oxperience. 
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The course covered the following topics: 



Adult Learning Principles 
Options for Teaching Methods 
Role of Evaluation in Learning 
Developing Effective Lesson Flens 
Accomodating Cultural Differences 
Barriers to Adult Learning 

The course was offered over a three day period to 102 
participants. CEI staff provided the bulk of the organizational 
and instructional staff with assistance from other members of the 
training collaborative. 

PROGRAM OBJECTIVES MET 

There are two general descriptions of objectives met in this 
report, the first is a summary written by each of the program 
directors and the second is the observations by the supervisors 
of the students at their workplace. Overall, the CEI workplace 
literacy and basic skills programs have a 70% completion rate 
overall, while the national average is about 30% for literacy 
programs; the close relationship between trainees and their 
counselors and the specific intervention which counselors often 
facilitate are one important factor in the number of adults which 
are retained in a given program. 



A. Program Co-ordinators Reports 

The general social, economic, and program goals were listed 
earlier. The specific quantifiable program accomplishments were 
as follows: 



1. Adult Basic Education 

Sixty-seven percent of the trainees who began the program in 
February 1989 completed the 39-week, 156 hour program in 
reading, writing and math. Of the remaining persons, 71% 
completed at least one 13 week module. In addition, 3 people 
joined classes in progress. Of these, 100% remained through 
the end of the program. On the average, ABE students entered 
reading below the 4th grade level. In i^'dition to the study, 
organization, dictionary, and test-taking skills acquired 
during the program, the trainees demonstrated an increase in 
comprehension as measured by the CEI Reading Comprehension 
Assessment Instrtoment. Trainees increased the volxime and 
quality of writing with more appropriate paragraph structure, 
as well as better grammar and spelling. 
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topics were rated with means over four. A single question asking 
how useful the entire seminar was, resulted in a mean score of 
4.19 out of a possible five. Ninety-one percent of the 
participants wrote that they would reconmiend the seminar to their 
col leagues . 

B. Overall Impact on Participants As Observed by Supervisors 

The supervisors of participants were asked to complete a 
questionnaire on each participant indicating where they observed 
improvement as a result of the Workplace Literacy Partnership 
program. The items on the questionnaire represented a wide range 
of areas including communication, self confidence, reading and 
writing skills, comprehension of directions given (and 
willingness to ask questions if directions are not understood), 
risk taking, and relations with co-workers. 



The Adult Basic Education Program (ABE) 



The mort often cited area of improvement for these students 
was greater self-confidence. The next area most noted by 
supervisors was in communications where they saw both an 
improvement in willingness to communicate and an increased 
ability to communicate in stressful situations. A few also 
observed improved reading and writing skills and greater 
participation in meetings. One supervisor saw improved risk 
taking and one saw an increased understanding of directions 
given. 



English as a Second Language (ESL) 



There were two areas where supervisors noted improvement for 
large numbers of participants: communication and self confidence. 
The next most noted areas of improvement were in reading and 
writing skills, speaking up when instructions were not 
understood, improved relations with co-workers and willingness to 
perform duties. Supervisors also saw in^rovement in 
understanding appropriated behavior in the workplace and a 
willingness to ask questions. One supervisor noted an increased 
understanding of staff and patient needs. 



Adult Diploma Program (ADP) 



For the participants in this program there were several 
areas that supervisors indicated improvement. These included 
stronger job performance, greater efficiency in the workplace due 
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English as a Second Language 



Nancy Centrella, Director 

CEI conducted five ESL programs in four nursing homes* Each 
course within a program focused on helping students develop the 
English skills necessary to perform their jobs efficiently and 
with i.ospect for the well-being of the residents in their care* 
Emphasis, too, was placed on transferring the skills developed in 
the workplace to everyday life. For the most part, participants 
in the program worked in Dietary and Housekeeping departments and 
as Nurses' Aides. 

Most of the classes within the ESL program reflected varying 
language competencies and work duties. Therefore, in addition to 
published texts, instructors developed supplementary materials to 
meet the language and work needs of each participant. 
Instructors and site coordinators also counseled participants. 

The following is a summary of each program by site: 



Vernon Hall Nursing Home 
ESL 520 

Four employee/participants completed the program. Participants 
entered the program with a Low Literacy - Beginning level of 
competencies in English. At the completion of the program, all 
four participants had progressed to Beginning - Advanced level in 
all competency areas (auditory discrimination, auditory 
vocabulary, grammar, sight words, reading and writing). (six 
participants enrolled in the program: two left the company). 



ESL 590 

Eight employee/participants completed the program. The 
Intermediate level course added a communication competency to 
those already described above. Participants progressed from a 
Beginning level - Intermediate/Advanced in the course of the 
program. 



Armenian Nursing Home 
ESL 530 



Five employee/participants completed the program. All 
participants made noticeable progress from Low Beginning - Low 
O Advanced levels, (seven participants enrolled: two withdrew for 
hKJC personal reasons). 

~~ 19 20 



Greenery Rehab Center 



ESL 540 

Five employee/participants completed the program. An additional 
five participants were transferred to an ABE program at the same 
site. Participants in the multi-level program advanced from, in 
some cases, fuctionally illiterate to beginning literacy; others 
from a Literacy - Intermediate level. (13 participants were 
enrolled: five were transferred to other classes, three withdrew 
for personal reasons). 



Westridge Center 
ESL 550 

Four employee/participants completed the program. All of the 
participants progressed from either Beginning - Intermediate or 
Intermediate - Advanced levels. (11 participants were enrolled: 
four (of the 11) were transferred to a bilingual class, three 
withdrew because of work-celated conflicts). 
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Assessment: 



Many problems centered around a^sescment. Initial assessment was 
spotty and incomplete at some sites. This was a result of a 
personnel problem. There were two issues. Since two key people 
were part-time and had limited time to spend at the sites, 
several participants missed some of the initial testing* 
Teachers were unable to do the assessments in class because of 
the limited time allowed by the sites, in addition to lack of 
time, there was a problem regarding the testing instruments. 
This was especially true with the math assessment. Some people 
took the ABE assessment instrument while others took the AOP 
assessment instrument.' 

At the end of the program some of the same issues occurred. All 
students read the same passages from CEI's Reading Assessment 
instrument. However, some students were tested for math skills 
with the ADP math test while others were retested with the ABE 
math test. The reason was that we were trying to see how many 
students we could recommend to go on to the ADP program. For 
this reason instructors decided to use the AOP exam. However, I 
was concerned about having a test/retest condition in order to 
satisfy evaluation requirements for the grant, so since Westridge 
was not .included in the proposal for the following year, and thus 
would not have an ADP program, it was decided to use the ABE 
instrument, and supplement it with some additional material which 
the instructor covared in class. 

The problem with this was that we do not have strong test/retest 
data for the Adult Basic Education program. For future ABE 
programs I strongly recommend using the ABE math test in the 
beginning and near the end of the program. The final exam could 
then be the ADP math assessment, depending on the progrc >s of the 
students, the material covered in class, and the amount of time 
devoted to math, it seems foolish to use the ADP math assessment 
if the material covered does not relate to the material on the 
assessment instrument* 

Tc assess reading comprehension, I strongly recommend using a 
standardized reading test such as the Stanford Diagnostic Reading 
Test (SORT), brown level, while students on the ABE level will 
invariably score low, if at the end of the class they are given 
the same test, some measurable progress can be determined. My 
research of the SORT indicates that the reliablity of using 
alternate forms for testing and retesting is not as high as it 
should be, hence I recommend that the same form be given. 

TO asses decoding skills, I recommend using the WRAT since it is 
easy to administer, and gives a good diagnostician a lot of 
information regarding students' ability to decode. It is easy to 
scor«^, and it provides another "number" to show quantifiable 
results. This is a useful test to show qualitative and 
quantifiable improvement. 
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Adult Diploma Program 
Lemuel Shattuck Hospital 
Final Report 
December, 1989 to March, 1990 
J\my McMahon, Diploma Co-ordinator 



Thirteen of twenty-one students (62%) completed their course 
work, graduated and were awarded diplomas. 

The students were a combined class from three sites: 
Armenian Nursing Home in Jamaica Plain, MA, the Greenery 
Rehabilitation and Skilled Nursing Center in Brighton, MA and the 
Lemuel Shattuck Hospital in Jamaica Plain, MA. 

Approximately 80% of the class was comprised of nursing 
assistants. Students ranged in age from 28 to 61 years of age, 
with an average age of 32 years old. Also, 75% of the class were 
women. Of those students enrolled under the Workplace Literacy 
Partnership, 90% were immigrants from liti, Jamaica and 
Barbados. 

Students in the Adult Disploma Program have made steady 
progress in both academic and job skills performances. To date, 
one student received a promotion 5.ince enrolling in the program. 
Many students intend on enrolling in some form of higher 
education next fall: five are interested in nursing school (two 
have already applied), three are interested in communxty college, 
two are interested in a trade/techuical school and two are 
undecided. One student has already been accepted to a community 
college for next fall. 



Appendices 
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A. Example of detailed reports filed 
by program directors 



B. External Evaluator Report 



FINAL REPORT 
WESTRIDGE HEALTHCARE CENTER 
ADULT BASIC EDUCATION 

Submitted by: Maxine Keats 

ABE Coordinator 
30 March 1990 

Introduction 

This final report will outline the Adult Basic Education Course 
held at the Hestridge Healthcare Center. The report will 
describe testing procedures, curriculum, class composition, and 
student progress. 

Assessment Procedures 

Several measures were uses to evaluate scudent progress. 

Initial Assessment included: 

Reading: Comprehension: Graded Passages - 3rd grade 

4th grade 

Mathematics: CEI's ABF math instrument 

Addition, Subtraction, Multiplication, 
Division, Fractions, Decimals, and 
Geometry 

Writing: An initial sample was taken to use for comparison 
at various points in the progiara. 

Final Evaluation included: 

Reading: Comprehension: Graded Passages - 3-13 grade level 

Math: CEI's ABE math instrument and/or CEI's ADP math 

instrument 

Writing: A final sample was taken to compare with earlier 
samples 



Throughout the modules instructor constructed tests were used to 
monitor progress and identify potential problems. Since 
standardized tests are not appropriate at this level we usr.* 
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test/retest strategies i!tilizing our own instruments for 
quantitative data* 

Writing samples done throughout our programs provide qualitative 
data* 



ABE Designated Writing Levels: 

Poor: writes words and attempts to write at sentence 
level 

Fair: writes sentences with some attention to word 

order r grammar, and punctuation; may attempt to - 
write at paragraph level 

Good: writes paragraphs showing some command of grammar 

and word order; attempts to use more sophisticated 
vocabulary 

Advanced: writes well-developed paragraphs with strong 
command of grammar and uses sophisticated 
vocabulary 



DESIGN OF PROGRAM 

Three modules meeting for 13 weeks for a total of 39 weeks. 



DESCRIPTION OF CLASS 

ABE Monday/Wednesday 2:00-4:00 pm 

Instructor: Catherine M. Smith 



Student Texts: 
Module I 



Module II 



Modeul III 



Reading Power 

Essential Mathematics for Life 

America's Story - Book I 
Essential Mathematics for Life 
Great American Stories - Book I 

America's Story - Book II 
Contemporary Mathematics - Book I 
Great American Stories - Book II 



CLASS COMPOSITION 

Five students were originally enrolled for this module, one 
joined class in progress but withdrew for reasons explained 
be low . 




Ir Carolyn 

2. Antonio 

3. Frances (Sue) 

4. Millicent (joined 4/13 withdrew 5/17 - lack of time) 
5* Jeanne 

6* Deann 
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This is a multi-level class, which requires ad^tional planning 
and supplementary materials. The range was from about second 
grade through high school. 

Progress was snown by all students, including the student who 
withdrew after the first module. This can be observed by 
examining the attached individual progress reports. 

It is important to note that Millie was a supervisor, 

and working at a very high level. She is clearly able to work 
at least on a junior college level. The problem was that the 
rest of the class was functioning well beneath her level. 
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Deann 



Initial Assessment: 



Reading: 3rd grade passage 60% 

4th grade passage - unable to do 

Math: ABE Math Assessment: 14/71 - 20% 

Writing: content: Poor 
form: Poor 
volume: 1 sentence 



Final Evaluation: 



Reading: 80% Primer 

50% 1st grade 
25% 3rd grade 

Writing: Content: Poor/Low Fair 
Form: Poor 



Math: ABE Math Assessment: 25/71 - 35% 

Improved skills working with whole numbers. 



Attendance: 64 out of 78 sessions 
Continuing Education Units: 12.8 



Narrative: Deann had difficulty following class discussions as 
well as following directions as a result of her 
severe learning problems. She feared exposing her 
weaknesses, so the instructor only requested 
participation when she felt certain that Deann could 
be successful. Deann has a low conceptual level, - 
and she is very sensitive about her deficiencies. 
She was thrilled that, following a class discussion 
about Alaska, she was able to tell her family many,., 
facts which were then corroborated by a National 
Geographic Special. The ABE class clearly had a 
powerful impact on her self-perception. During the 
course of the literacy program, Deann was promoted 
from PCA I to PCA II. while in-house staff training 
was a factor, according to Bernadette Oinenen, the 
Staff Development Coordinator, CEI's program made a 
considerable contribution because it helped develop 
Deann's confidence. Given her limitations Deann 
showed slight improvement in math and writing. 
However, the course clearly improved her self-iroage 
and increased her confidence. Deann* s reading 
skills did not improve appreciably; however, her 
math skills improved 75% according to the CEI ABE 
math assessment instrument, and her writing volume 
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Supervisor 



Comments unavailable 

Deann was promoted from a PCA to a PCA II 



increased appreciably (see attached sample) • 



Instructor : 



Student: 



Deann has made a real effort to improve her skills^*... 
but continues to need support at the ABE level. 
Her writing shows improvement in productivity and 
her willingness to write shows improvement. She 
has become more responsive and is willing to risk 
error because she feels comfortable in the group. 
She is particularly eager to complete the history -A 
book on her own. In math she has difficulty 
following exercises having more than one-*step, but 
she works at rote skills. Deann has problems with. . 
memorizing. Deann began to study before exams in 
subjects she liked and eventually began to 
volunteer. She became quite motivated. Her self- ., 
confidence began to develop, but skills remain at 
very low level. She is unable to work 
independently and finds group work difficult, but 
makes attempts to follow. 

Except for the direct quote, the following 
comments were extrapolated from the checklist 
Deann completed: 

I have improved in history and math. I have more 
confidence since I've taken this class. My math 
and writing have improved. Reading is still very 
hard. '•I like my class they are nice to work with 
and the teacher . 



'J" 
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CONTIhJUING 
EDUCATION 
INSTITUTE INC, 



IL -.1??!.*:^'^.'^ AVENUE. NEEOHAM. MA OS194 JBOl ^Q-ABOS 



ADULT DIPLOMA PROGRAM 
WRITING ASSESSMENT 



Name: 



Date: //^ 7/^4 ^ 

|M20§E ONE (1) of the following topics and develop your Ideas about that 
topic, using complete sentences and paragraph structure: 

^ Tell why you want tc continue your education. 
2. Identify and describe the country you come from. 

^* ItTli^"^ '''"'^f.r ^ '^"^ ^'hy you 

did or did not like it. ^ 

a 
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CONTINUING 
EDUCATION 

INSTITUTE INC. 163 highland avenue, needham. ma 03194 (S17J 449-4aos 



Workplace Literacy Partnership 
for Nursing Home Employees 



External Evaluation Report 



Marcus G, Lieberman, Ph.D. 



1990 



Overview 



This report describes five major areas of evaluation 
activities during the 1989-1990 academic year of the project* 
The five areas are the following: 

(1) orientation 

(2) observations 

(3) consultations on measurement and evaluation 

(4) quantitative results and 

( 5 ) r ec ommendat i on s 

The evaluator would like to express great appreciation to 
the project staff and the director of CEI for their time, 
patience and cooperation over the course of the past year* He 
recognizes the concerns the staff have, both' for the students and 
for the need for an effective evaluation and the compromises that 
are necessary to achieve optimal results in both areas* 



1* Orientation 
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In the fall of 1989, the evaluator interviewed staff from 
each of the three programs, the Adult Diploma Program (ADP), the 
Adult Basic Education Program (ABE) and the English as a Second 
Language Program (ESL). He also interviewed the Project Director 
an-a the Director of the Continuing Education Institute (CEI). He 
reviewed the project proposal for goals and educational 
methodology. Each of the programs provided curriculum materials 
and the evaluation measures presently on hand. They also 
provided schedules for their courses, a description of the 
student population and copies of their credentials. 

CEI has been conducting educational programs in the 
workplace for a number of years and has continued to revise its 
materials and tests to meet the needs of the emp' ■^yees it serves. 
There is a great deal of emphasis on diagnostic measures to 
inform the instructors of the strengths and weaknesses of 
participants in reading, writing, grammar and .nathematics . The 
teaching is intended to respond to each student's needs. End of 
unit tests are criterion-referenced to match the unit's 
objectives. The instructors have created, with consultation from 
each program director, worksheets, activities and tests for each 
course taught. The experienced staff were able to describe the 
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perspective of potential adult students who are often very self- 
conscious about their English and described the various ways they 
have found that provide a minimum- risk situation where the 
students can converse in English without fear of ridicule. The 
observations conducted by the evaluator are described in the next 
section . 



2. Observation of ESL and ABE Classes 



During November and December, 1989, the evaluator. Dr. 
Marcus G. Lieberman, visited two ESL and one ABE classes. All 
three classes were held in the nursinc? homes where the students 
worked. 

In Vernon Hall, in Cambridge ard at the Greenery, in Boston, 
the classes were held in a room apart from the patient area. In 
Armenian, the class was held in one of the patient's rooms while 
she was at an activity. Each of the classes took two hours. 

All three instructors shared their lesson plans for the day 
with the evaluator and before the class began, described the 
range of level of the students, both in achievement and in 
motivation. 

During the classes, all three instructors showed a high 
degree of warmth, enthusiasm, sensitivity and competence. The 
instruction ranged from concepts in language and mathematics to 
actual applications of what they were learning to their nursing 
home duties and responsibilities. In one ESL class, the 
instructor had brought articles of loose clothing so that the 
students could describe in English what they were doing as they 
put on or took off the clothing from themselves or each ether. 
The aides often assist the nursing home patients in dressing or 
undressing and they were told that if they explained to the 
patients what they were about to do, anxiety and confusion would 
be reduced. 

The instructors used workbooks, papers prepared to practice 
the concepts presented, role play and conversation to reinfotw- 
knowledge and skills presented. 



3. Consultation on Measurement and Evaluation 
Achievement Tests 

A large proportion of the evaluator's time was spent 
^ discussing the importance of evaluation instruments and data 

EEiC 34 35 



apart from the 'diagnostic measures employed by the instructors* 
Project directors were extremely reluctant to administer a pre- 
test based on the objectives for a course or unit. While 
participants were eager to learn English , they were very shy 
about expressing themselves publicly. The program directors felt 
that asking the students to take a pre-test would set a tone 
early in a very sensitive course that would likely result in 
students dropping out of the course and not returning for a 
second and subsequent classes. 

Some time ago, the Director of CEI supported an effort to 
review the existing adult basic skills measures for validity to 
the CEI instructional program and sufficient ''evels of relevant 
reliability coefficients. None ol the measui s reviewed were 
judged satisfactory in both areas by the CEI aff. 

The Director recognizes the need for some standardized 
yardstick to demonstrate the amount and nature of improvement of 
CEI's clients. The results could be used for both evaluation and 
marketing purposes. However, the staff feel that each individual 
comes with different levels of skills in a variety of areas and 
the general nature of the existing standardized measures does not 
permit the investigation of specific strengths and weaknesses 
addressed by the instruction. 

The staff are reluctant to use standardized tests designed 
for elementary age school children for the adults enrolled in 
their programs because the examples and applications are often 
inappropriate. In addition, the norms for these tests, based on 
children's responses to the items would also be inappropriate. 

The ESL and ABE staff have created measures and begun 
research to determine a coriesponderce between grade level and 
test score so they can measure growch due to the program. The 
ADP staff use midterms and final ey.ams throughout their coutse to 
determine whether or not students pass each module, but do not 
pre-test the students to assess p'^ior knowledge. 

Each instructor keeps detailed records on their students and 
logs each objective met or skill acquired. Therefore, there are 
specific reports on each stuuent as to growth accomplished, but 
it is impossible to summarize the results into a simple 
descriptive statement. 

Supervisory Observations 

In addition to student-administered achievement tests, 
supervisors are asked to complete a short survey to document 
changes or areas of improvement they have noticed in students. 
The first draft of this measure included several areas that 
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supervisors were not likely to observe, such as specific English 
writing or oral skills. A revised version includes many 
behaviors more likely to be observed by supervisors. The 
supervisor indicates whether or not he or she has noticed 
improvement in each of several areas. 



4. Quantitative Results 

Since many of the instruments were developed while the 
courses were being taught, quantitative results are sketchy this 
first year. 

In the ESL program, assessments were made in the areas of 
Grammar and Reading for one of the groups late in the year. 
There were nine levels in each area: Literacy 1, Literacy 2, Low 
Beginning, Beginning, High Beginning, Low Intermediate, 
Intermediate, High Intermediate and Advanced. Pour of seventeen 
students (24%) remained at the same level in Grammar and thirteen 
cf seventeen (76%) increased their level. In Reading, four of 
twenty-one students tested (19%) remained at their original 
level, sixteen of twenty-one (76%) increased their level and one 
(5%) decreased in level. 

In addition to these achievement scores, perceptions of 
supervisors as to improvement by students in various areas were 
also recorded. The accompanying table shows the numbers of 
students for whom supervisors completed evaluation sheets 
describing their perceptions of improvement (or the lack thereof) 
in several area^. 

The number of students evaluated is given at the top of each 
column and numbers in the columns show the number of students for 
whom supervisors saw improvement. 



ERIC 



37 

36 



English as a Second Language Workplace Education 

Svinunary of Supervisors' Evaluations 
Modules I and II 

Percentages of students Where Improvement Was Observed 
Module I: 



Site: 



Numbers of Students 

Increased Willingness To 
Communicate At Work 

Ability To Communicate In 
Problematic or Stressful 
Situations 

Improved Reading and 
Writing Skills 

Level of Self Confidence 

Program Meeting Initial 
Expectations (yes) 



Armenian 
Mid Final 



(5) 
80 

40 

40 

40 
80 



(3) 
iOO 

67 

33 

33 
67 



Greenevy 
Final 
(7) 
28 

43 

14 

57 
57 



vernon Hall 
Mid Final 
(3) (2) 



100 
100 

67 

S3 
33 



50 
50 

50 

100 
0 
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Module II: 



Numbers of Studer g (5) (4) (7) 

Increased Willingness To 100 25 ICO 

Communicate At Work 

Ability To Communicate In 40 25 lOO 

Problematic or Stressful 

Situations 

Improved Reading and 60 0 72 

Writing Skills 

Level of Self Confidence 100 50 loo 

Program Meeting Initial 20 0 86 

Expectations (yes) 
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Adult Diploma Program 



Summary of Quantitative Results 



Tables 1 and 2 present the results of the administration of tests to 
students in Oanuary and May, 1989 in the areas of Mathematics and Reading. 

In the area of mathematics, two students took a lower level test in 
January, so their results cannot be compared with the test they took in 
May. The other 12 students took the same level test at bbth times. In all 
four areas measured by the tests, (1) Number Theory/Graphs and Charts (2) 
Computation/Fractions/Decimals (3) Percent and Formulae and (4) 
Metrics/Geometry the increase from January to May was statistically 
significant (p< . 01) . 

In the area of reading, there were three sets or levels of material 
for the students to read and answer multiple choice questions which spanned 
the levels of the Bloom Taxonomy. The three levels correspond to 4th 
grade, 7th grade, and 9 to 11th grade. At each level students were given 
two passages, one from a non-fiction piece and the other, a selection from 
a real work of literature. 

In January, students took either the Level I or Level II passages, b; " 
by May, all students were administered the Level III passages with the 
results shown in Table 2. Eight of 14 students received a score of 80 ox- 
higher on the Facts passage and eight of 14 received a score of 80 or 
higher on the Literature passage. 

In Table 1, the data for two students who took different tests are 
presented iirst, though no real comparison is possible. Then the results 
for the other twelve students are presented. The numbers in pa entheses 
under the test headings are the numbers of items in each section. 
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Table 1 

Mathematics Results for 
Individual Students 
January • May, 1989 

January May 

Number 

Add Sub Mult Div Frac Dec Geom Theory Fractions Pet Metrics 

No. of Grph/Cht Decimals Form Geometry 

Items (12) (12) (12) (12) (12) (12) (12) (13) (14) (8) (16) i 

Mean 10. 5 10 ♦58. 5 9 3. 55. 5 0 11 9 5.5 8 



Num Thry Comp/Prac Percent Metrics 
Grph/Cht Dec Formulae Geom 
(13) (14) (8) (16) 

* * * * 

Mean 5.33 9.00 1.91 r.OO 10.67 12^00 6.55 11.58 

* Significant Difference from January score. (p<.01) 



Table 2 

Reading Results for 
Adult Diploma Program St':dents 
January - May, 1985 

N = 14 

(Mean Percent Score) 
January May 

Level I Level II Level ill 

Pacts Lit. Facts Lit. Facts Lit, 



^ Mean 100 100 77.9 81.4 
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79.3 76.4 



Summary of Responses to Questionnaires 
By Participants at the 
Continuing Education Institute 
Nursing Assistants Instructor Course 

August 22-24, 1989 



Participants were asked to complete a questionnaire to assist 
the program staff in placing the most emphasis on the areas most 
needed. The respondents were to place a check under one of the two 
categories 'Understand/Use' or 'Need To Learn.' The results of the 
95 registrants are summarized in Table 1. 



Table 1 

Responses to Needs Assessment 
(Percent agreeing) 

Understand/ Need To 
Use Learn 

42.9 57.1 
27.8 72.2 

38-9 61.1 



41.9 58.1 

58,3 41.7 

48.3 51.7 

70.9 29.1 

68.6 31.4 

51.7 48.3 



29.3 70.7 

55.8 44.2 

21.3 78.7 

27.2 72.8 



47.1 52.9 

25.6 74.4 

17.0 83.0 
14.4 85.6 

13. 1 86.7 



Adult Learning Principles 

Building on adult experience 
Barriers to adult learning 
Special problems in nurses' 
aides population 

Options For Teaching Methods 
Participative lecture 
Demonstration and practice 
Role Play 
Discussion 
Small groups 
Audio- visual aids 

Role of Evaluation in Learning 
Identifying training needs 
Recognizing accomplishments 
Identifying areas for remediation 
Use of self-^ peer- and 

instructor-driven evaluation 

Effective Lesson Plans 

Identifying outcome goals 
Breaking down big tasks into 

small skills 
Selecting learning activities 

to teach skills 
Sequencing learning activities 
Integrating evaluation into 

the lesson plan 
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After the first day of the course, 97 participants responded 
to an evaluation form which allowed them to (1) rate itiaterial as 
to whether or not it was useful to meet the needs of trainees and 
(2) rate the presentation as to whether or not it was focused and 
easy to follow. Table 2 shows the percentage agreeing that the 
material was useful and that the presentation on the material was 
focused • 



Table 2 



Percentage Agreeing That 
Materials and Presentations Were 
Useful and Focused 



Material 
Useful 



Presentation 
Focused 



Introduction and overview of 
seminar/Chara'^teristics of adult 
learners 



95.6 



81.2 



How to setup a 75 hour nursing 
assistant training course 



93.6 



87,9 



How to identify and select outcome 
goals 



83.1 



66*7 



Assessing needs and evaluating 
outcomes 



93.5 



p:.8 



Focus Group 



83.1 



77.0 
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Page 3 

Table 3 shows the proportion of those respondents who rated 
each of the materials areas as USEFUL who also rated the 
PRESENTATION focused and easy to follow. 



Table 3 

Percentage of Those Who 
Rated the Materials as Useful 
Who Also Rated the Presentation as Focused 



Introduction and overview of seminar/ 

Characteristics of Adult Learners 83.5 

How to set up a 75 hour nursing assistant 

training course 90.6 

How to identify and select outcome goals 75.4 

Assessing needs and evaluating outcomes 95.1 

Focus Group 94.2 



After the second day of the course, respondents completed an 
evaluation form similar to day 1, but on new topics. Tables 4 and 
5 show the results of these evaluations. 
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Table 4 

Percentage Agreeing That 
Materials and Presentations Were 
Useful and Focused 



Material 

Useful 



Presentation 
Focused 



Accomodating cultural differences 
Building on adult experience 
Simplifying materials 
Developing effective lessons 
Focus group 



100.0 
100.0 
96.4 
69.0 
81.3 



96.5 
96.4 
93.9 
54.3 
79.7 



Table 5 

Percentage of Those Who 
Rated the Materials as Useful 
Who Also Rated the Presentation as Focused 



Accomodating cultural differences 
Building on adult experience 
Simplifying materials 
Developing effective lessons 
Focus Group 



96.5 
96.4 
96.2 
80.0 
96.6 
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After the third day of the course, respondents completed an 
evaluation form similar to days 1 and 2, but on new topics. Tables 
6 and 7 show the results of these evaluations. 



Table 6 

Percentage Agreeing That 
Materials and Presentations Were 
Useful and Focused 



Introduction 

Bcirriers to adult learning 
Accommodating learning differences 



Material 
Useful 

84.9 

97.9 

99.0 



Presentation 
Focused 

93.5 

99.0 

100.0 



Table 7 

Percentage of Those Who 
Rated the Materials as Useful 
Who Also Rated the Presentation as Focused 

Introduction 97.4 

Barriers to adult learning 100.0 

Accommodating learning differences 100.0 



Finally, on the third day, participants were asked to complete 
an evaluation form for the entire workshop. Ratings ranged from 
one (not at all comfortable with the material) to five (extremely 
comfortable with the material). Table 8 gives the mean ratings for 
each item. 



Table 8 

Mean Ratings For Items 
On the Conference Evaluation Form 

(5 Point Scale) 
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Adult Learning Principles 

Building on adult experience 4.65 

Barriers to adult learning 4.54 

Special problems in the nursing asst. pop. 4.38 

Options For Teaching Methods 

Participative lecture 4.55 

'demonstration and practice 4.53 

Hole play 4.28 

Discussion 4.56 

Small groups 4.53 

Audio-visual aids 4.55 

Role of Evaluation in Learning 

Identifying training needs 4.14 

Recognizing accomplishments 4.44 

Identifying areas for remediation 4,05 

Use of self-, peer-, and inst. dir. evaluation 4.24 

Effective Lesson Plans 

Identifying outcome goals 4.12 

Breaking down big tasks into small skills 4 12 

Selecting learning activities to teach skills 4.18 

Sequencing learning activities 4.17 

Integrating evaluation into lesson 4.18 



At the end of the? conference evaluation form, participants 
responded tc two final questions. On a scale from 1 (not at all 
useful) to 5 (very useful), how useful was the entire three day 
seminar? The mean rating for all participants was 4.19. 

The second question asked whether or not participants would 
^ . recommend the seminar to their colleagues. Ninety-one percent of 
. hKJC ^he respondents said they would. 



Summary 

On the application formr registrants reported their greatest 
needs in the areas o£ evaluation and lesson plans. Three areas 
where over 80% of the respondents indicated a need to learn were 
selecting and sequencing learning activities, and integrating 
evaluation into the lesson plan. 

During the first day, participants indicated that three of 
five areas were very useful, characteristics of adult learners, 
how to set up a training course, and assessing needs and 
evaluating outcomes. The most focused presentation was that on 
assessing needs and evaluating outcomes. The least focused 
presentation was on identifying and selecting outcome goals. 

During the second day, every participant rated accommodating 
cultural differences and building on adult experience as useful. 
Developing effective lessons was judged the least useful. Only 
developing effective lessons and the focus group were judged less 
focused. 

All the topics on the third day were judged useful and the 
presentations focused. 

When participants were asked to rate the entire three days 
on a scale from one to five (five being best), all topics were 
rated with means over four. A single question asking how useful 
the entire seminar was, resulted in a mean of 4.19 out of a 
possible 5.00. Ninety-one percent of the participants said they 
would recommend the seminar to their colleagues. 
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5. Recomnendations 

One major area lacking in the evaluation of these programs 
is a control or comparison group. In order to attribute change 
in reading, writing and mathematics skills to the program, It is 
necessary to show that there is little or no change in these 
skills without the program. Recognising that asking people to 
complete lengthy tests without providing feedback or intervention 
borders on unethical practice, some form of control may still be 
possible to implement. Often, there are several groups which 
meet during the year. Perhaps one group may be pre and post 
tested several months later (equal to the length of the program) 
and then later offered the course. 

The second major area requiring attention is instrument 
development, normj.ng, validation and reliability estimation. 
Over the course of the year, the evaluator met with personnel 
from each of the programs to discuss the various possibilities 
for measurement and evaluation. Program directors were very 
concerned that if potential students were given pretests, they 
were likely to find the experience intin.idating and not come back 
for the courses. However, they realized that pretests are 
necessary to show change. Some suggestions that arose out of the 
discussions included giving pretests early in the course, but 
after rapport had been established and the purpose of the test 
presented, most likely as a place to start and not to highlight 
embarrassing weaknesses in English or mathematics. By the end of 
the year, the staff in the ESL and ABE programs were working to 
create grade level material for students to read and to answer 
multiple choice comprehension test items on the reading. The ADP 
staff continues to refine their instruments which reflect the 
goals of their courses. 
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